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Addressing the needs of beginning readers 

This What Works Clearinghouse (WWC) topic review addresses a 
variety of programs and products developed to address the needs 
of beginning readers. For the current wave of reports, we focused on 
replicable programs or products for students in the early elementary 
settings (that is, grades K-3) including: core reading curricula, 
programs, or products to be used as supplements to other reading 
instruction, programs that focus on staff development, and literacy 
software. The What Works Clearinghouse (WWC) review on begin- 
ning reading focuses on reading interventions for students in kin- 
dergarten through grade 3 intended to increase skills in alphabetics, 
reading fluency, comprehension, or general reading achievement. 

Because there are so many reading interventions and studies in 
Beginning Reading, the What Works Clearinghouse set priorities 
for programs to be reviewed first. They included those that, on 
initial screening, had studies with the strongest (most rigorous) 
designs and those that, on initial screening, had the most studies.'' 

We looked at 887 studies of 153 programs that qualified for 
our review. Of these, 51 studies of 24 programs met our evidence 
standards, 27 without reservations and 24 with reservations.^ 

The remaining 129 programs had no studies that met the WWC 
evidence screens. Of these, 92 programs had one or more stud- 
ies that were reviewed and did not meet WWC evidence screens. 
Thirty-seven programs did not have any outcomes studies. 

In looking at the four outcome domains for the 24 interven- 
tions, 10 interventions had positive effects or potentially positive 



effects in all the outcome domains addressed in their studies 
(see table 1). Eleven interventions had a combination of positive 
or potentially positive effects in one or two domains while having 
mixed, negative, or no discernible effects in other domains. 

Three had only mixed effects or no discernible effects across 
domains. 

Intervention ratings for Beginning Reading 

Each beginning reading program reviewed had at least one 
study meeting WWC standards (with or without reservations) and 
received a rating of effectiveness in one or more of the four out- 
come domains (alphabetics, fluency, comprehension, and gen- 
eral reading achievement). The rating is designed to characterize 
the existing evidence, taking into account: quality of the research 
design, statistical significance of the findings, size of the differ- 
ence between participants in the intervention and comparison 
conditions, and consistency in findings across studies. 

The research evidence can be rated as positive, potentially posi- 
tive, mixed, no discernible effects, potentially negative, or negative 
(see the WWC Intervention Rating Scheme) . Table 1 shows the 
effectiveness ratings for the 24 beginning reading programs in the 
four outcome domains (empty cells indicate that no evidence was 
reported). Table 2 lists the programs for which there were no stud- 
ies meeting WWC evidence screens. This includes interventions 
with no studies and interventions with outcomes studies that were 
reviewed but did not meet WWC evidence screens. 



1. Thirty-two additional interventions (involving 36 quasi-experimental design studies) passed the initial screening criteria but were not included in this 
wave of Beginning Reading reviews. These interventions were those that on initial screening had only one eligible study that met WWC evidence stan- 
dards with reservations (i.e., had the fewest numbers of studies, which also used less rigorous designs). 

2. Seven additional single-case studies have dispositions pending. The WWC is currently developing standards for the review of single-case studies. 
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Table 1 Effectiveness ratings for 24 beginning reading interventions in four domains 



Alphabetics 


Fluency 
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General reading 
achievement 
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Rating of 


Extent of 


effectiveness 


evidence^ 


effectiveness 


evidence' 


effectiveness 


evidence' 


effectiveness 


evidence' 



Accelerated Reader/Readinq Renaissance 

(http://www. ren learn, com/reading, htm) 

Auditory Discrimination in Depth/ 

Lindamood Phonemic Sequencing ®^ 

ClassWide Peer Tutoring (http://www.jgcp.ku.edu) 

Cooperative Integrated Reading and Composition ® (CIRC) 

(http://www.successforall.net/elementary/readingwings.htm) 

Corrective Reading (http://www.sraonline.com) 

Daisy Quest ^ 

Early Intervention in Reading ® (EIR) 

(http://www. earlyinterventioninreading. com) 

Earobjcs® (http://www.earobics.com) 

Failure-Free Reading (http://www. failurefreeonline. com/index_parents.php) 
Fast ForWord ® (http://www.scilearn.com) 

Fluency Formula ™ (http://www.scholastic.com/fluencyformula) 

Kaplan SpellRead (http://kaplank12.com) 

Ladders to Literacy (http://www. brookespublishing. com) 

Little Books (http://www.goodyearbooks.com) 

Peer-Assisted Learning Strategies (PALS) (http://kc.vanderbilt.edu/pals) 
Read Naturally (http://www.readnaturally.com/) 

Read, Write, Tvpe ^" (http://www. talkingfingers. com) 
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Table 1 Effectiveness ratings for 24 beginning reading interventions in four domains (continued) 



Alphabetics 


Fluency 


Comprehension 


Generai reading 
achievement 


Rating of 


Extent of 


Rating of 


Extent of 


Rating of 


Extent of 


Rating of 


Extent of 


effectiveness 


evidence^ 


effectiveness 


evidence' 


effectiveness 


evidence' 


effectiveness 


evidence' 



Reading Recovery® (httD://www.readinarecoverv.ora) 




Small 


<S> 


Small 


(S> 


Small 


Start Makino a Reader Todav® (httD://www.aetsmartoreaon.ora) 




Small 


<St 


Small 


(S> 


Small 


SteoDino Stones to Literacy (httD://www.soDriswestcom) 




Small 










Success for All® (httD://www.successforall.net) 


<s> 


Medium 
to large 






e 


Medium 
to large 


Voyaoer Universal Literacv Svstem® MD://www.vovaaer!earnina.com) 


<s> 


Medium 
to large 






0 


Small 


Waterford Early Readina Proaram^"" 

(http://www.pearsondigital.com/waterford) 




Small 






O 


Small 


Wilson Readina System® (httD://www.wiisonianauaae.com) 


(S> 


Small 


Q 


Small 


Q 


Small 




Medium 
to large 



Medium 
to large 



Note: The WWC intervention reports describe each program and provide information on the students, the cost, and the scope of use. To view the intervention reports, piease ciick on the program name or go to www.whatworks.ed.aov . 
Foiiowing each program name is the deveioper’s or distributor’s website address. The research evaiuated addresses some but not aii grade ieveis targeted by these interventions. Grade ieveis are reiated to student age and may affect 
outcomes. For a comparison of targeted grade ieveis and grade ieveis in the studies reviewed by the WWC, see Appendix A2. 



1. A rating of “medium to iarge” requires at ieasttwo studies and two schoois across studies in one domain and a totai sampie size across studies of at ieast 350 students or 14 ciassrooms. Otherwise, the rating is “smaii.” 

2. There is no singie website iisted as the product is soid by a number of distributors. See intervention report for further detaiis. 

3. There is no website iisted as the deveioper distributes the product individuaiiy. See intervention report for further detaiis. 



O Positive effects: 
strong evidence of 
a positive effect 
with no overriding 
contrary evidence 




Potentiaiiy 
positive effects: 
evidence of a 
positive effect 
with no overriding 
contrary evidence 




Mixed effects: 
evidence of 
inconsistent effects 




No discernibie 
effects: no 
affirmative evidence 
of effects 




Potentiaiiy 
negative effects: 
evidence of a 
negative effeot 
with no overriding 
contrary evidence 



O Negative effects: 
strong evidence of 
a negative effect 
with no overriding 
contrary evidence 
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Average improvement indices for each domain 

The WWC computes an average improvement index for each domain and each 
study, as well as a domain average improvement index across studies of the same 
intervention (see the Technicai Detaiis of WWC-Conducted Computations^ . 

The improvement index represents the difference between the percentile rank 
of the average student in the intervention condition and the percentile rank of 
the average student in the comparison condition. It can take on values between 
-50 and +50, with positive numbers denoting results favorable to the intervention 
group. Unlike the rating of effectiveness, which is based on four factors, the 
improvement index is based only on the size of the difference between the inter- 
vention and the comparison conditions.^ 



Figure 1 Alphabetics: average improvement 
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* Medium to large extent of evidence (see appendix A1). 



Alphabetics 

The alphabetics domain includes skills tied to word decoding and word recogni- 
tion. Alphabetics comprises five constructs: 

• Phonemic awareness (or phoneme awareness) refers to the understanding that 
the sounds of spoken language-phonemes-work together to make words, and 
phonemes can be substituted and rearranged to create different words. 

• Phonological awareness is a more encompassing concept and refers to aware- 
ness of larger spoken units such as syllables and rhyming words. 

• Letter Identification refers to knowledge of the names of the letters of the alpha- 
bet and has been shown to be a predictor of reading development. 

• Print awareness refers to knowledge or concepts about print and awareness of 
common characteristics of books. 

• Phonics refers to the ability to associate letters and letter combinations with 
sound and blending them into syllables and words. 

We reviewed alphabetics outcomes for 18 beginning reading programs, and the 
average improvement index ranged from +1 to +36 percentile points (figure 1). 

Fluency 

Fluency is the ability to read connected text accurately, automatically, and with 
expression, while still extracting meaning from it. Many fluency outcomes rely 
on measuring both speed and accuracy. We reviewed fluency outcomes for 11 
beginning reading programs, and the average improvement index ranged from +2 
to +46 percentile points (figure 2). 

Comprehension 

The comprehension domain includes measures in two constructs both oriented 
toward understanding the meaning of what is read. The first, vocabulary develop- 
ment, refers to the development of knowledge about the meanings, uses, and 
pronunciation of words. Measures of both receptive (listening) and expressive 
(spoken or written) vocabulary were included. The second construct, reading 
comprehension, refers to the understanding of the meaning of a passage and the 
context in which the words occur. We reviewed comprehension outcomes for 19 
programs, and the average improvement index ranged from -15 to +20 percentile 
points (figure 3). 



3. To enable comparisons across interventions, improvement indices are calculated from student-level findings. In the case of Ladders to Literacy in the alphabetics domain, the average improve- 
ment index does not represent all the findings reviewed by the WWC, as some of the findings were reported on the classroom or school level, and student-level improvement indices could not be 
computed. For further details please see Technical Details of WWC-Conducted Computations . 
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Figure 2 Fluency: average improvement 
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Figure 4 General reading achievement: average improvement 
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* Medium to large extent of evidence (see appendix A1). 
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Figure 3 Comprehension: average improvement 




General reading achievement 

The general reading achievement domain includes outcomes that either combine 
two or more of the previous domains (aiphabetics, reading fiuency, and compre- 
hension) or provide some other type of summary score, such as a “totai reading 
score” on a standardized reading test. We reviewed general reading achievement 
outcomes for 5 programs, and the average improvement index ranged from -fIO to 
+32 percentiie points (figure 4). 
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Table 2 Programs reviewed with no studies meeting WWC evidence screens'* 



100 Book Challenge (http://www.100bookchallenge.com) 

Academic Associates Learning Centers® (http://www.academic-associates.com/index.htm) 

Academy of Reading (no website avaiiabie) 

Alpha-Time (no website avaiiabie) 

AlphabiTunes (http://www.alphabitunes.com) 

America’s Choice (http://www.americaschoice.org) 

Athen’s Tutorial Program (http://www.uga.edu/~atp) 

Balanced Early Literacy Initiative (no website avaiiabie) 

Barton Reading & Spelling System (http://www.bartonreading.com) 

Benchmark Word Recognition Program (http://www.benchmarkschool.org) 

Book Buddies (http://www.readwritethink.org/lessons/lesson_view.asp?id=160) 

Bookmark (no website available) 

Bradley Reading and Language Arts (no website available) 

Breakthrough to Literacy (http://www.breakthroughtoiiteracy.com) 

Bridge (http://vmw. utpress. utoronto. ca/GCentre/0778402762. html) 

Bring the Classics to Life (http://www.edconpublishing.com) 

California Early Literacy Learning (CELL) (http://www.ceil-exll.com) 

Carbo Reading Styles Program (http://www.nrsi.com) 

CIERA School Change Project (http://www.ciera.org) 

C.L.A.P., A sound Approach to Pre-Reading Skills (http://www.soundreading. 
com/pages/programs.cfm?id=9B2CEECC-ABD3-4584-BBD283CA5DA431C3) 

CompassLearning (http://www.compasslearning.com) 

Compensatory Language Experiences and Reading Program (CLEAR) (no website available) 

Comprehensive Curriculum for Early Student Success (ACCESS) 

(http://www.sharingsuccess.org/code/eptw/profiles/ndn1.html) 

Concept Phonics Fluency Set (http://www.oxtonhouse.com/concept_phonics.htmi) 
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Letter People (http://www.abramsandcompany.com/ietterpeopleJndex.cfm) 

Letterland (http://www.ietterland.com) 

Leap into Phonics (http://www.leapintolearning.com/products.htmi) 

LeapFrog SchoolHouse (http://www.leapfrogschoolhouse.com) 

LinguiSystems (http://www.linguisystems.com) 

Literacy Collaborative (http://literacycoliaborative.org) 

Literacy First (http://www.iiteracyfirst.com) 

LocuTour Multimedia Cognitive Rehabilitation (http://www.locutour.com) 

Merit Reading Software Program (http://www.meritsottware.com) 

My Reading Coach'*’"" (http://www.myreadingcoach.com/nrp/fluency.html) 

National Geographic Society and Arizona Geographic Alliance K-8 program 

(http://alliance.la.asu.edu/geoliteracy/GeoLitNCSS.pdf) 

New American Schools (no website available) 

New Century Integrated Instructional System (http://www.ncecorp.com) 

New Heights (no website avaiiabie) 

North Carolina A+ Schools network (http://aplus-schools.uncg.edu) 

Onward to Excellence (http://www.nwrel.org/scpd/ote) 

Pacemaker (http://www.agsglobe.com/group.asp?nGrouplnfolD=a0822451026) 

Pause Prompt & Praise (http://www.peta.edu.au/ 
Teaching_resources/Teaching_Tips/page_1559.aspx) 

Peabody Language Development Kits 

(http://ags.pearsonassessments.com/group.asp?nGroupinfolD=a8550) 

Performance Learning Systems (http://www.plsweb.com) 

Programmed Tutorial Reading (no website avaiiabie) 

Project FAST (Families Are Students and Teachers) (http://www.wcer.wisc.edu/EAST) 

Project LISTEN’S Reading Tutor (http://www.cs.cmu.edu/%7Eiisten/index.html) 

(continued) 
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Table 2 Programs reviewed with no studies meeting WWC evidence screens"* (continued) 



Cornerstone Literacy Initiative (http://www.comerstoneiiteracy.org) 


Project LISTEN’S Writing Tutor (no website available) 


Crossties (no website avaiiabie) 


Project PLUS (Partnership Linking University School Personnel) (no website available) 


Davis Learning Strategies® Program (http://www.davisiearn.com) 


Project Read (http://www.projectread.com) 


Destination Reading (http://www.riverdeep.net/portai/ 


QuickReads (http://quickreads.org) 


page?_pageid=336,1&_dad=portai&_schema=PORTAL) 


Rainbow Reading Program (http://www.rainbowreading.co.nz) 


Different Ways of Knowing (http://www.differentways.org/gaief/different_ways.htmi) 


Read Well (http://store.cambiumlearning.com) 


Direct Instruction/DISTAR (no website avaiiabie) 


Reading Intervention for Early Success (http://www.eduplace.com/intervention/readintervention) 


Direct Instruction/Horizons (http://www.sraoniine.com) 


Reading Rods (http://www.etacuisineaire.com/readingrods.readingrods.jsp) 


Direct Instruction/RITE (http://www.ritemaii.com) 


Reading Speed Drills (http://www.oxtonhouse.com/reading_speed_drills.html) 


Direct Instruction/Spelling Mastery (http://mcgraw-hiii.co.uk/sra/readingmastery.htm) 


Reading Success from the Start (no website available) 


Direct Instruction/SRA (http://www.sraoniine.com) 


Reading Theater (http://playbooks.com/index.shtml) 


Direct Instruction/Teacher Your Child to Read in 100 Easy Lessons (no website avaiiabie) 


Reading Together''''^ (http://leamingtogether.com/inschool/readingtogether.html) 


Direct, Intensive, Systematic, Early, and Comprehensive 


Reading Upgrade (http://www.learningupgrade.com) 


(DISEC) Instruction (no website avaiiabie) 


Right Start to Reading (no website available) 


Discover Intensive Phonics for Yourself (http://www.readinghorizons.com) 


Road to the Code (http://www.brookespublishing.com/store/books/blachman-4382/indexhtm) 


Dr. Cupp Readers® & Journal Writers (http://www.cindycupp.com) 


SAIL (Second grade Acceleration in Literacy) (no website available) 


Edison Schools (http://www.edisonschools.com) 


Saxon Phonics (http://saxonpublishers.harcourtachieve.com) 


Emerging Readers Software (http://www.soundreading.com) 


Schoolwide Early Language and Learning (SWELL) (no website available) 


Essential Skills Software (http://www.essentialskilis.net) 


SkillsTutor (http://www.achievementtech.com) 


Evidence Based Literacy Instruction (http://www.ebli.org) 


Soar to Success (http://www.edupiace.com) 


Fast Track Action Reading Program (no website avaiiabie) 


Sunday System (http://www.sondaysystem.com) 


Felipe’s Sound Search (no website available) 


Sound Field System (no website available) 


First grade Literacy Intervention Program (FLIP) (no website available) 


Sound Foundations (no website available) 


Flippen Reading Connections'*’''" (http://www.flippengroup.com/education/index.html) 


Sound Partners (http://www.wri-edu.org/partners/sound-partners.htm) 


Frontline Phonics (http://www.frontlinephonics.com) 


Sound Reading (http://www.soundreading.com/srs_new/index.cfm) 


Fundations (http://www. fundations. com) 


Sounds Abound (http://www.linguisystems.com) 



(continued) 
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Table 2 Programs reviewed with no studies meeting WWC evidence screens'* 


(continued) 


Funnix (http://www.funnix.com) 


Sounds and Symbols Early Reading Program (http://ags.pearsonassessments.com) 


GOcabulary Program for Elementary Students (http://www.gocabulaiy.com/gocabel.htm) 


S.PJ.R.E. (http://www.epsbooks.com/dynamic/catalog/program.asp?seriesonly=3250M) 


Goldman-Lynch Language Simulation Program (no website availabie) 


Starfall (http://www.starfail.com) 


Goldman-Lynch Sounds-in-Symbols Development Kit 

(http://ags.pearsonassessments.com/group.asp?nGroupinfoiD=highhat) 


STEPS (Sequential Teaching of Explicit Phonics and Spelling) 

(http://www.stepsreadingcenter.com) 


Stories and More (http://brightbluesoftware.com/stories.htm) 


Guided Discovery LOGD (no website avaiiable) 


Story Comprehension to Go (http://www.linguisystems.com) 


Hooked on Phonics® (http://secure.hop.com/index.cfm) 


Strategies that Work (no website available) 


HOTS (http://www.hots.org) 


Student Teams Achievement Divisions (STAD) (http://www.pwcs.edu/curriculum/sol/stad.htm) 


Huntington Phonics (no website availabie) 


SuccessMaker® Reading (http://www.pearsondigital.com) 


IntelliTools Reading (http://www.intellitools.com) 


Sullivan Program (no website available) 


Invitations to Literacy (http://www.eduplace.com/rdg/itl) 


Voices Reading (http://www.zaner-bloser-voices.com/reading) 


Irlen Method (http://irten.com) 


VoWac (Vowel Oriented Word Attack Course) (http://www.vowac.com/OLD%20PAGES/default.oid.htm) 


Jigsaw Classroom (http://www.jigsaw.org) 


Wiggleworks (http://teacher.scholastic.com/products/wiggleworks/index.htm) 


Johnny Can Spell (http://www.nine-enterprises.eom/nine/navOO.asp) 


WORKSHOP WAY — Instant Personality Phonics Activities 


Jostens Integrated Language Arts Basic Learning System (no website available) 


(http://www.workshopway.org/workshop-way.htm) 


Kindergarten Works (no website available) 


Wright Group’s Intervention Program (http://www.wrightgroup.com) 


Kindergarten Intervention Program (KIP) (no website available) 


Writing to Read (http://brightbluesoftware.com/wtr.htm) 



4. The table includes all eligible programs with no studies and all eligible programs with no studies meeting evidence standards. The list was compiled trom interventions that were suggested tor WWC review by public submission and 
interventions that arose in studies found during the WWC literature search. The list is not inclusive of all reading interventions, and some of the interventions may no longer be actively distributed. 



For more information about the studies reviewed and the WWC methodoiogy, piease see the WWC Beginning Reading Technicai Appendices . 



WWC Topic Report Beginning Reading 



August 13, 2007 



8 





Appendix 



Appendix A1 Extent of evidence for the aiphabetics, fluency, comprehension, and generai reading achievement domains 



Intervention name 


Number of studies 


Sample size (schools/students) 


Extent of evidence 


Aiphabetics 








Accelerated Reader/Reading Renaissance 


0 


0 


na 


Auditory Discrimination in Depth/Lindamood Phonemic Sequencing® 


1 


5/146 


Small 


Cooperative Integrated Reading and Composition® 


0 


0 


na 


Corrective Reading 


1 


8/over 70 


Small 


ClassWide Peer Tutoring 


0 


0 


na 


Daisy Quest 


3 


nr/187 


Small 


Early Intervention in Reading® 


1 


2/56 


Small 


Earobics® 


2 


4/104 


Small 


Failure-Free Reading 


1 


8/93 


Small 


Fast ForWord® 


3 


5/295 


Small 


Fluency Formula^™ 


0 


0 


na 


Kaplan SpellRead 


2 


over 9/139 


Small 


Ladders to Literacy 


4 


over 14/760 


Moderate to large 


Little Books 


0 


0 


na 


Peer-Assisted Learning Strategies 


3 


17/295 


Small 


Read Naturally 


0 


0 


na 


Read, Write, Type’^™ 


1 


5/146 


Small 


Reading Recovery® 


3 


over 14/226 


Small 


Start Making a Reader Today® 


1 


6/84 


Small 


Stepping Stones to Literacy 


2 


17/120 


Small 


Success for All® 


7 


67/3,103 


Moderate to large 


Voyager Universal Literacy System® 


3 


14/719 


Moderate to large 


Waterford Early Reading Program^'” 


1 


6/76 


Small 



(continued) 
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Appendix A1 Extent of evidence for the aiphabetics, fluency, comprehension, and generai reading achievement domains (continued) 



Intervention name 


Number of studies 


Sample size (schools/students) 


Extent of evidence 


Wilson Reading System® 


1 


8/71 


Small 


Fluency 








Accelerated Reader/Reading Renaissance 


0 


0 


na 


Auditory Discrimination in Depth/Lindamood Phonemic Sequencing® 


0 


0 


na 


Cooperative Integrated Reading and Composition® 


0 


0 


na 


Corrective Reading 


1 


8/over 70 


Small 


ClassWide Peer Tutoring 


0 


0 


na 


Daisy Quest 


0 


0 


na 


Early Intervention in Reading® 


0 


0 


na 


Earobics® 


1 


1/74 


Small 


Failure-Free Reading 


1 


8/93 


Small 


Fast ForWord® 


0 


0 


na 


Fluency Formula^" 


1 


5/128 


Small 


Ladders to Literacy 


1 


over 1/66 


Small 


Kaplan SpellRead 


2 


over 9/139 


Small 


Little Books 


0 


0 


na 


Peer-Assisted Learning Strategies 


3 


5/295 


Small 


Read Naturally 


2 


2/106 


Small 


Read, Write, Type™ 


0 


0 


na 


Reading Recovery® 


1 


nr/74 


Small 


Start Making a Reader Today® 


1 


6/84 


Small 


Stepping Stones to Literacy 


0 


0 


na 


Success for All® 


0 


0 


na 


Voyager Universal Literacy System® 


0 


0 


na 


Waterford Early Reading Program^” 


0 


0 


na 


Wilson Reading System® 


1 


8/71 


Small 


Comprehension 








Accelerated Reader/Reading Renaissance 


1 


nr/178 


Small 



(continued) 
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Appendix A1 Extent of evidence for the aiphabetics, fluency, comprehension, and generai reading achievement domains (continued) 



Intervention name 


Number of studies 


Sample size (schools/students) 


Extent of evidence 


Auditory Discrimination in Depth/Lindamood Phonemic Sequencing® 


1 


5/146 


Small 


Cooperative Integrated Reading and Composition® 


2 


over 8/702 


Moderate to large 


Corrective Reading 


1 


8/over 70 


Small 


ClassWide Peer Tutoring 


0 


0 


na 


Daisy Quest 


0 


0 


na 


Early Intervention in Reading® 


1 


2/57 


Small 


Earobics® 


0 


0 


na 


Failure-Free Reading 


1 


8/93 


Small 


Fast ForWord® 


3 


over 11/292 


Small 


Fluency Formula^™ 


1 


5/128 


Small 


Kaplan SpellRead 


2 


over 9/139 


Small 


Ladders to Literacy 


3 


over 6/489 


Moderate to large 


Little Books 


0 


0 


na 


Peer-Assisted Learning Strategies 


2 


6/99 


Small 


Read Naturally 


1 


1/94 


Small 


Read, Write, Type’^™ 


1 


5/146 


Small 


Reading Recovery® 


2 


nr/156 


Small 


Start Making a Reader Today® 


1 


6/84 


Small 


Stepping Stones to Literacy 


0 


0 


na 


Success for All® 


6 


65/2,565 


Moderate to large 


Voyager Universal Literacy System® 


2 


6/321 


Small 


Waterford Early Reading Program^'” 


1 


6/76 


Small 


Wilson Reading System® 


1 


8/71 


Small 


General reading achievement 








Accelerated Reader/Reading Renaissance 


1 


nr/394 


Small 


Auditory Discrimination in Depth/Lindamood Phonemic Sequencing® 


0 


0 


na 


Cooperative Integrated Reading and Composition® 


0 


0 


na 


Corrective Reading 


0 


0 


na 
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Appendix A1 Extent of evidence for the aiphabetics, fluency, comprehension, and generai reading achievement domains (continued) 



Intervention name 


Number of studies 


Sample size (schools/students) 


Extent of evidence 


ClassWide Peer Tutoring 


1 


6/218 


Small 


Daisy Quest 


0 


0 


na 


Early Intervention in Reading® 


0 


0 


na 


Earobics® 


0 


0 


na 


Failure-Free Reading 


0 


0 


na 


Fast ForWord® 


0 


0 


na 


Fluency Formula^™ 


0 


0 


na 


Kaplan SpellRead 


0 


0 


na 


Ladders to Literacy 


0 


0 


na 


Little Books 


1 


6/314 


Small 


Peer-Assisted Learning Strategies 


0 


0 


na 


Read Naturally 


0 


0 


na 


Read, Write, Type’^™ 


0 


0 


na 


Reading Recovery® 


5 


over 14/452 


Moderate to large 


Start Making a Reader Today® 


0 


0 


na 


Stepping Stones to Literacy 


0 


0 


na 


Success for All® 


6 


31/1,767 


Moderate to large 


Voyager Universal Literacy System® 


0 


0 


na 


Waterford Early Reading Program^'” 


0 


0 


na 


Wilson Reading System® 


0 


0 


na 



na = not applicable/not studied 
nr = not reported 

Note: A rating of “moderate to large” requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 
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Appendix A2 Targeted populations 



Intervention name 


Targeted students (grades) 


Students in studies reviewed (grades) 


Accelerated Reader/Reading Renaissance 


All levels 


K-3 


Auditory Discrimination in Depth/Lindamood Phonemic Sequencing® 


K-12 


1 


Cooperative Integrated Reading and Composition® 


2-8 


3 


Corrective Reading 


3-9 


3 


ClassWide Peer Tutoring 


K-6 


1 


Daisy Quest 


PK-2 


K-2 


Early Intervention in Reading® 


K-6 


1 


Earobics® 


PK-3 


K-3 


Failure-Free Reading 


K-12 


3 


Fast ForWord® 


PK-12 


K-3 


Fluency Formula^™ 


1-6 


2 


Kaplan SpellRead 


K-12 


1-3 


Ladders to Literacy 


K 


K 


Little Books 


K-12 


K 


Peer-Assisted Learning Strategies 


K-12 


1-3 


Read Naturally 


1-8 


1-2 


Read, Write, Type^“ 


K-3 


1 


Reading Recovery® 


1 


1 


Start Making a Reader Today® 


K-2 


1 


Stepping Stones to Literacy 


PK-K 


K 


Success for All® 


PK-8 


K-3 


Voyager Universal Literacy System® 


K-3 


K 


Waterford Early Reading Program^"" 


K-2 


K 


Wilson Reading System® 


2-12 


3 



Note: This table presents a comparison of targeted grade ieveis and the grade ieveis in the studies reviewed by the WWC. Grade ieveis are reiated to student age and may affect outcomes due to 
differences in the students’ deveiopmentai stages as weii as differences in schooi size and organization. 
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Appendix A3.1 



Summary of statistically significant^ or substantively important^ positive findings 



Positive findings 


Intervention name Alphabetics 


Fluency 


Comprehension 


General reading achievement 


Accelerated Reader/Reading Renaissance 








Ross, Nunnery, & Goldfeder, 2004 na 

(randomized controiied trial) 


na 


ns 


STAR Early Literacy test 


Auditory Discrimination in Depth/Lindamood Phonemic Sequencing 








Torgesen et ai., 2003 [ADD i/s. ns 

Read, Write, Type intervention] 

(randomized controiied trial) 


na 


ns 


na 


Torgesen et ai., 2003 [ADD vs. regular CTOPP: Phoneme Elision Subtest; 

instruction] (randomized controiied trial) CTOPP Phoneme Segmenting Subtest; 


na 


ns 


na 



Woodcock Reading Mastery Test: Word 
Attack Subtest; Woodcock Reading 
Mastery Test: Word Identification Subtest 



Cooperative integrated Reading and Compositiorf’ 










Bramlett, 1994 (quasi-experimentai design) 


na 


na 


ns 


na 




Skeans, 1991 (quasi-experimentai design) 


na 


na 


ns 


na 




Corrective Reading 












Torgesen etai., 2006 


WRMT-R: Word Identification Subtest; 


Oral Reading Fluency 


ns 


na 




(randomized controiied trial) 


TOWRE: Sight Word Efficiency Subtest 










ClassWide Peer Tutoring 

Greenwood et ai., 1993 
(randomized controiied trial) 


na 


na 


na 


ns 




DaisyQuest 

Baker &Torgensen, 1995 [DaisyQuest 


Undersea Challenge; Production 


na 


na 


na 




vs. Hint and Hunt software] 
(randomized controiied trial) 


Test of Segmenting 










Baker & Torgensen, 1995 [DaisyQuest i/s. 


Undersea Challenge; Production 


na 


na 


na 




math software] (randomized controiied trial) 


Test of Segmenting 










Foster et ai., 1994 [Experiment 1: Child- 


Phonological Awareness Test 


na 


na 


na 




care Facility] (randomized controiied trial) 


(PAT) (b); Screening Test of 
Phonological Awareness: 
Experimental Version (STCPA-E) 










(continued) 
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Appendix A3.1 



Summary of statistically significant^ or substantively important^ positive findings (continued) 









Positive findings 




Intervention name 


Alphabetics 


Fluency 


Comprehension 


General reading achievement 


Foster et al., 1994 [Experiment 
2: Kindergarten Ciassrooms] 
(randomized controlled trial) 


Undersea Challenge; Production Test of 
Segmenting; Production Test of Blending 


na 


na 


na 


Mitchell & Fox, 2001 [DaisyQuest I's 
teacher-delivered phonological awareness 
instruction] {mtiom'ized controlled trial) 


ns 


na 


na 


na 


Mitchell & Fox, 2001 [DaisyQuest 
I'S other software programs group] 
(randomized controlled trial) 


Phonological Awareness 
Test (PAT) (a): total 


na 


na 


na 


Early Intervention in Reading 










Taylor, Frye, Short, & Shearer, 1991 
(randomized controlled trial) 


Segmentation and blending; 
Vowel sounds 


na 


ns 


na 


Earobicsf^ 










Cognitive Concepts, 2003 
(randomized controlled trial) 


ORAL-J: Blending into Words 
Subtest; ORAL-J: Segmenting into 
Sounds; ORAL-J: Rhyming Words 


ns 


na 


na 


Valliath, 2002 (quasi-experimental design) 


CTOPP: Sound Matching 


na 


na 


na 


Failure-Free Reading 










Torgesen et at, 2006 
(randomized controlled trial) 


ns 


ns 


ns 


na 


Fast ForWorrP 

Borman & Benson, 2006 
(randomized controlled trial) 


na 


na 


ns 


na 


Scientific Learning Corporation, 
2005a (randomized controlled trial) 


TOPA: Phonological Awareness Subtest; 
TOPA: Letter Sounds Subtest 


na 


na 


na 


Scientific Learning Corporation, 
2005b (randomized controlled trial) 


ns 


na 


na 


na 


Scientific Learning Corporation, 
2005c (randomized controlled trial) 


na 


na 


Degrees of Reading Power 


na 


Scientific Learning Corporation, 
2006 (randomized controlled trial) 


ns 


na 


na 


na 



(continued) 
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Appendix A3.1 



Summary of statistically significant^ or substantively important^ positive findings (continued) 









Positive findings 




Intervention name 


Alphabetics 


Fluency 


Comprehension 


General reading achievement 


Overbay & Baenen, 2003 
(quasi-experimental design) 


na 


na 


ns 


na 


Fluency Formula™ 

Sivin-Kachala & Bialo, 2005 
(randomized controlled trial) 


na 


ns 


ns 


na 


Kaplan SpellRead 










Rashotte, MacPhee, & Torgesen, 
2001 (randomized controlled trial) 


CTOPP: Blending Words Subtest; 
CTOPP: Segmenting Words Subtest; 
TOWRE: Phonetic Decoding Efficiency 
Subtest; WRMT-R: Word Attack Subtest 


ns 


GORT-3: Comprehension Subtest; 
WDRB: Comprehension Subtest 


na 


Torgesen etat, 2006 
(randomized controlled trial) 


TOWRE: Phonetic Decoding Efficiency 
Subtest; WRMT-R: Word Attack Subtest 


ns 


ns 


na 


Ladders to Literacy 










O’Connor, 1999 (Study A: Intensive 
Professional Development) 
(quasi-experimental design) 


Short Term Memory; Segmentation; 
Blending; Woodcock Johnson 
Tests of Achievement: Letter- 
Word Identification Subtest 


na 


ns 


na 


O’Connor, 1999 (Study B: Traditional 
Professional Development) 
(quasi-experimental design) 


Segmentation 


na 


ns 


na 


O’Connor et at, 1996 (quasi- 
experimental design) 


ns 


ns 


ns 


na 


Fuchs et al., 2001 (randomized controlled 
trial with randomization problems) 


ns 


na 


na 


na 


Little Books 










Phillips, Norris, Mason, & Kerr, 1990 
(randomized controlled trial) 


na 


na 


na 


ns 


Peer-Assisted Learning Strategies 

Fuchs, Fuchs, Kazdan, & Allen, 

1999 (randomized controlled trial 
with randomization problems) 


na 


na 


Stanford Diagnostic Reading Test 
III: Reading Comprehension 


na 



(continued) 
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Appendix A3.1 



Summary of statistically significant^ or substantively important^ positive findings (continued) 



Positive findings 


Intervention name 


Alphabetics 


Fluency 


Comprehension 


General reading achievement 


Mathes & Babyak, 2001 (randomized 
controlled trial with randomization problems) 


ns 


ns 


na 


na 


Mathes, Howard, Allen, & Fuchs, 
1998 (quasi-experimental design) 


ns 


ns 


na 


na 


Mathes, Torgesen, Ciancy-Minchetti 
et at, 2003 [Comparison #1: PALS 
i/s. Usual Reading Curriculum Group] 
(quasi-experimental design) 


CTOPP Phonemic Segmentation; 
WRMT: Word Attack Subtest 


ns 


ns 


na 


Mathes, Torgesen, Ciancy-Minchetti 
et at, 2003 [Comparison #2: PALS 
i/s. Teacher-Directed Instruction 
Group] (quasi-experimental design) 


ns 


ns 


ns 


na 


Read Naturally 










Hancock, 2002 (randomized controlled trial) 


na 


ns 


ns 


na 


Mesa, 2004 (quasi-experimental design) 


na 


ns 


na 


na 


Read, Write & Type! 










Torgesen et at, 2003 [Read, Write 
& Type! I/s. ADD intervention] 
(randomized controlled trial) 


ns 


na 


ns 


na 


Torgesen et at, 2003 [Read, Write 
& Type! vs. Regular instruction] 
(randomized controlled trial) 


CTOPP Phoneme Segmenting 
Subtest; Woodcock Reading Mastery 
Test: Word Attack Subtest 


na 


ns 


na 


Reading Recoverf 

Baenen etal., 1997 
(randomized controlled trial) 


na 


na 


na 


ns 


Pinnell, DeFord, & Lyons, 1988 
(randomized controlled trial) 


Observation Survey: Concepts 
about Print Subtest 


na 


CTBS: Reading Comprehension 
Subtest; CTBS: Reading 
Vocabulary Subtest 


Observation Survey: Dictation 
Subtest; Observation Survey: 
Writing Vocabulary Subtest 


Pinnell etal., 1994 
(randomized controlled trial) 


na 


na 


na 


Gates-MacGinitie; Observation 
Survey: Dictation Subtest; Woodcock 
Reading Mastery Test-Revised 



(continued) 
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Appendix A3.1 



Summary of statistically significant^ or substantively important^ positive findings (continued) 







Positive findings 




Intervention name 


Alphabetics 


Fiuency 


Comprehension 


Generai reading achievement 


Schwartz, 2005 
(randomized controlled trial) 


Observation Survey: Concepts 
about Print Subtest; Observation 
Survey: Word Recognition Subtest 


S0RT-R3; Observation Survey: 
Text Reading Level Subtest 


ns 


Observation Survey: Dictation 
Subtest; Observation Survey: 
Writing Vocabulary Subtest 


Iverson & Tunmer, 1993 
(quasi-experimental design) 


Phoneme Deletion Task; Yopp-Singer 
Phoneme Segmentation Test; 
Observation Survey: Goncepts about 
Print Subtest; Observation Survey: 
Letter Identification Subtest; Dolch 
Word Recognition Test; Observation 
Survey: Word Recognition Subtest; 
Pseudoword Decoding Task 


na 


na 


Observation Survey: Dictation 
Subtest; Observation Survey: 
Writing Vocabulary Subtest 


Start Making a Reader Todaf 


Baker, Gersten, & Keating, 2000 
(randomized controlled trial) 


Woodcock Reading Mastery Test- 
Revised: Word Identification Subtest 


Oral Reading Fluency First-Grade 
Passage; Oral Reading Fluency 
Second-Grade Passage 


ns 


na 


Stepping Stones to Literacy 


Nelson, Benner, & Gonzalez, 2005 
(randomized controlled trial) 


CTOPP: Phonological Awareness; 
DIBELS: Phoneme Segmentation 
Fluency; DIBELS: Initial Sound Fluency; 
DIBELS: Letter Naming Fluency; 
DIBELS: Nonsense Words Fluency 


na 


na 


na 


Nelson, Stage, Epstein, & Pierce, 2005 
(randomized controlled trial) 


CTOPP: Phonological Awareness; 
DIBELS: Letter Naming Fluency; 
WRMT-R: Word Identification Subtest; 
WRMT-R: Word Attack Subtest 


na 


na 


na 


Success for AiP 


Borman et at, 2006 
(randomized controlled trial) 


WRMT: Word Identification Subtest; 
WRMT: Word Attack Subtest 


na 


WRMT: Passage 
Comprehension Subtest 


na 


Dianda & Flaherty, 1995 
(quasi-experimental design) 


ns 


na 


ns 


ns 


Madden et at, 1993 
(quasi-experimental design) 


ns 


na 


ns 


ns 
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Appendix A3.1 



Summary of statistically significant^ or substantively important^ positive findings (continued) 









Positive findings 




Intervention name 


Alphabetics 


Fluency 


Comprehension 


Generai reading achievement 


Ross, Alberg, & McNelis, 1997 
(quasi-experimental design) 


ns 


na 


ns 


ns 


Ross & Casey, 1998 
(quasi-experimental design) 


ns 


na 


ns 


ns 


Ross et at, 1998 
(quasi-experimental design) 


ns 


na 


ns 


ns 


Smith etal., 1993 
(quasi-experimental design) 


ns 


na 


ns 


ns 


Voyager Universal Literacy Systenf 










Frechtling, Zhang, and Silverstein, 
2006 (quasi-experimental design) 


ns 


na 


na 


na 


Hecht, 2003 (quasi-experimental design) 


ns 


na 


ns 


na 


Hecht & Torgesen, 2002 
(quasi-experimental design) 


ns 


na 


ns 


na 


Waterford Early Reading Program™ 

Hecht & Close, 2002 
(quasi-experimental design) 


ns 


na 


ns 


na 


Wilson Reading 










Torgesen et al., 2006 
(randomized controlled trial) 


TOWRE: Phonetic Decoding Efticiency 
Subtest; WRMT-R: Word Attack Subtest 


ns^ 


ns^ 


na 



na = not studied 

ns = not statistically significant 

nsi = not substantively important 



1 . According to WWC criteria, if a program finds a stafistically significant effect, there is less than a 5% chance that this difference is due to chance. The level of stafistical significance was calculated by the WWC and, where necessary, 
corrects for clusfering within classrooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . For the formulas the WWC used to calculate the statistical signifi- 
cance, see the Technical Details of WWC-Conducted Computations . 

2. For rating purposes, the WWC considered the statistical significance of the findings and the magnitude of the effect, also called the effect size. An average effect size is the sum of all the effect sizes of the student outcomes in a study in 
a single domain divided by the number of those outcomes. The WWC considers an average effect size across all student outcomes in one study in a given domain to be substantively important if it is equal to or greater than 0.25. 

3. The fluency, comprehension, and vocabulary components of Wilson Reading Systenf were eliminated from instruction at the request of Torgeson et al. for the purposes of the study. For further information about the program implement- 
ed, please see the research and findings sections of the Wilson Reading Svstem® - reoort . 
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Appendix A3.2 Summary of statistically significant^ or substantively important^ average effect across outcomes by domain 



Average effect across outcomes 


Intervention name 


Alphabetics 


Fluency 


Comprehension^ 


General reading achievement 


Accelerated Reader/Reading Renaissance 

Ross, Nunnery, & Goldfeder, 2004 


na 


na 


ns. Substantively important 


Statistically significant. 


(randomized controlled trial) 








Substantively important 


Auditory Discrimination in Depth/Lindamood Phonemic Sequencing 








Torgesen et al., 2003 [ADD i/s. Read, Write, 


Statistically significant. 


na 


ns, nsi 


na 


Type intervention] (randomized controlled trial) 


Substantively important 








Torgesen et al., 2003 [ADD I's. Reguiar 


Statistically significant. 


na 


ns, nsi 


na 


instruction] (mtiom'izeii controlled trial) 


Substantively important 








Cooperative integrated Reading and Compositioif 








Bramlett, 1994 (quasi-experimental design) 


na 


na 


ns, nsi 


na 


Skeans, 1991 (quasi-experimental design) 


na 


na 


ns, nsi 


na 


Corrective Reading 

Torgesen et al., 2006 (randomized 


ns, nsi 


Statistically significant, 


ns, nsi 


na 


controlled trial) 




Substantively important 






ClassWide Peer Tutoring 

Greenwood et at, 1993 
(randomized controlled trial) 


na 


na 


na 


ns. Substantively important 


DaisyQuest 

Baker STorgensen, 1995 [DaisyQuest 


Statistically significant. 


na 


ns, nsi 


na 


i/s. Hint and Hunt software] 
(randomized controlled trial) 


Substantively important 








Baker & Torgensen, 1995 [DaisyQuest vs. 


Statistically significant. 


na 


ns, nsi 


na 


math software] (randomized controlled trial) 


Substantively important 








Foster et al., 1994 [Experiment 1: Chiid-care 


Statistically significant. 


na 


na 


na 


Fac///(y/ (randomized controlled trial) 


Substantively important 








Foster et al., 1994 [Experiment 2: 


Statistically significant. 


na 


na 


na 


Kindergarten Classrooms] 
(randomized controlled trial) 


Substantively important 








Mitchell & Fox, 2001 [DaisyQuest I's. 
teacher-delivered phonological awareness 
instruction] {mtiom'aeii controlled trial) 


ns, nsi 


na 


na 


na 


Mitchell & Fox, 2001 [DaisyQuest 
i/s other software programs group] 
(randomized controlled trial) 


ns, nsi 


na 


na 


na 
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Appendix A3.2 Summary of statistically significant^ or substantively important^ average effect across outcomes by domain (continued) 



Average effect across outcomes 


Intervention name 


Alphabetics 


Fluency 


Comprehension^ 


General reading achievement 


Early Intervention in Reading 

Taylor, Frye, Short, & Shearer, 1991 


Statistically significant, 


na 


ns. Substantively important 


na 


(randomized controlled trial) 


Substantively important 








Earobic^ 

Cognitive Concepts, 2003 
(randomized controlled trial) 


ns, Substantively important 


ns, nsi 


na 


na 


Valliath, 2002 (quasi-experimental design) 


ns, Substantively important 


na 


na 


na 


Failure-Free Reading 

Torgesen et at, 2006 
(randomized controlled trial) 


ns, nsi 


ns, nsi 


ns. Substantively important 


na 


Fast ForWorrP 

Borman & Benson, 2006 
(randomized controlled trial) 


na 


na 


ns, nsi 


na 


Scientific Learning Corporation, 2005a 
(randomized controlled trial) 


Statistically significant, nsi 


na 


na 


na 


Scientific Learning Corporation, 2005b 


Statistically significant, 


na 


na 


na 


(randomized controlled trial) 


Substantively important 








Scientific Learning Corporation, 2005c 


na 


na 


Statistically significant, 


na 


(randomized controlled trial) 






Substantively important 




Scientific Learning Corporation, 2006 
(randomized controlled trial) 


ns, nsi 


na 


na 


na 


Overbay & Baenen, 2003 
(quasi-experimental design) 


na 


na 


ns 


na 


Fluency Formula™ 

Sivin-Kachala & Bialo, 2005 


na 


ns, Substantively important 


ns. Substantively important 


na 


(randomized controlled trial) 






negative effect 




Kaplan SpellRead 

Rashotte, MacPhee, & Torgesen, 
2001 (randomized controlled trial) 


ns, Substantively important 


ns, nsi 


ns, nsi 


na 


Torgesen et at, 2006 (randomized 


Statistically significant, 


ns, Substantively important 


Statistically significant, 


na 


controlled trial) 


Substantively important 




Substantively important 




Ladders to Literacy 

O’Connor, 1999 (Study A: Intensive 


Statistically significant. 


na 


ns, Substantively important 


na 


Professional Development) 
(quasi-experimental design) 


Substantively important 









(continued) 
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Appendix A3.2 Summary of statistically significant^ or substantively important^ average effect across outcomes by domain (continued) 



Average effect across outcomes 


Intervention name 


Alphabetics 


Fluency 


Comprehension^ 


General reading achievement 


O’Connor, 1999 (Study B: Traditional 


Statistically significant. 


na 


ns, nsi 


na 


Professional Development) 
(quasi-experimental design) 


Substantively important 








O’Connor et at, 1996 
(quasi-experimental design) 


ns. Substantively important 


ns. Substantively important 


ns, nsi 


na 


Fuchs et al., 2001 


ns, na'* 


na 


na 


na 


Little Books 

Phillips, Norris, Mason, & Kerr, 1990 
(randomized controlled trial) 


na 


na 


na 


ns. Substantively important 


Peer-Assisted Learning Strategies 

Fuchs, Fuchs, Kazdan, & Allen, 


na 


na 


Statistically significant, 


na 


1999 (randomized controlled trial 
with randomization problems) 






Substantively important 




Mathes & Babyak, 2001 (randomized 


Statistically significant, 


ns. Substantively important 


na 


na 


controlled trial with randomization problems) 


Substantively important 








Mathes, Floward, Allen, & Fuchs, 


Statistically significant. 


ns. Substantively important 


na 


na 


1998 (quasi-experimental design) 


Substantively important 








Mathes, Torgesen, Clancy-Minchetti 
et al., 2003 [Comparison #1: PALS 
vs. Usual Reading Curriculum Group] 
(quasi-experimental design) 


ns. Substantively important 


ns, nsi 


ns, nsi 


na 


Mathes, Torgesen, Clancy-Minchetti 
et al., 2003 [Comparison #2: PALS vs. 
Teacher-Directed Instruction Group] 
(quasi-experimental design) 


ns. Substantively important 


ns, nsi 


ns, nsi 


na 


Read Naturally 

Flancock, 2002 (randomized controlled trial) 


na 


ns, nsi 


ns, nsi 


na 


Mesa, 2004 (quasi-experimental design) 


na 


ns, nsi 


na 


na 


Read, Write & Type! 

Torgesen et al., 2003 [Read, Write & Type! vs. 
ADD Intervention] (randomized controlled trial) 


ns, nsi 


na 


ns, nsi 


na 


Torgesen et al., 2003 [Read, Write 
& Type! vs. Regular instruction] 
(randomized controlled trial) 


ns, nsi 


na 


ns, nsi 


na 
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Appendix A3.2 Summary of statistically significant^ or substantively important^ average effect across outcomes by domain (continued) 



Average effect across outcomes 


Intervention name 


Alphabetics 


Fluency 


Comprehension^ 


General reading achievement 


Reading Recover^ 

Baenen etal., 1997 
(randomized controlled trial) 


na 


na 


na 


ns, nsi 


Pinnell, DeFord, & Lyons, 1988 


ns, Substantively important 


na 


Statistically significant. 


Statistically significant, 


(randomized controlled trial) 






Substantively important 


Substantively important 


Pinnell etal., 1994 


na 


na 


na 


Statistically significant, 


(randomized controlled trial) 








Substantively important 


Schwartz, 2005 


Statistically significant. 


Statistically significant. 


ns, nsi 


Statistically significant, 


(randomized controlled trial) 


Substantively important 


Substantively important 




Substantively important 


Iverson & Tunmer, 1993 


Statistically significant, 


na 


na 


Statistically significant, 


(quasi-experimental design) 


Substantively important 






Substantively important 


Start Making a Reader Todaf 

Baker, Gersten, & Keating, 2000 


Statistically significant. 


Statistically significant. 


ns, Substantively important 


na 


(randomized controlled trial) 


Substantively important 


Substantively important 






Stepping Stones to Literacy 

Nelson, Benner, & Gonzalez, 2005 


Statistically significant, 


na 


na 


na 


(randomized controlled trial) 


Substantively important 








Nelson, Stage, Epstein, & Pierce, 


Statistically significant, 


na 


na 


na 


2005 (randomized controlled trial) 


Substantively important 








Success for AiF 










Borman et at, 2006 


Statistically significant. 


na 


Statistically significant, nsi 


na 


(randomized controlled trial) 


Substantively important 








Dianda & Flaherty, 1995 (quasi-experimental design) 


ns. Substantively important 


na 


ns, Substantively important 


ns, Substantively important 


Madden etal., 1993 (quasi-experimental design) 


ns. Substantively important 


na 


na 


ns, Substantively important 


Ross, Alberg, & McNelis, 1997 
(quasi-experimental design) 


ns, nsi 


na 


ns, nsi 


ns, nsi 


Ross & Casey, 1998 
(quasi-experimental design) 


ns, nsi 


na 


ns, nsi 


ns, nsi 


Ross et at, 1998 (quasi-experimental design) 


ns. Substantively important 


na 


ns, nsi 


ns, nsi 


Smith etal., 1993 (quasi-experimental design) 


ns. Substantively important 


na 


ns, nsi 


ns, Substantively important 


Voyager Universat Literacy Systenfi 

Frechtling, Zhang, and Silverstein, 

2006 (quasi-experimental design) 


ns. Substantively important 


na 


na 


na 



(continued) 



WWC Topic Report Beginning Reading 



August 13, 2007 



23 





Appendix A3.2 Summary of statistically significant^ or substantively important^ average effect across outcomes by domain (continued) 



Average effect across outcomes 


Intervention name 


Alphabetics 


Fluency 


Comprehension^ 


General reading achievement 


Hecht, 2003 (quasi-experimental design) 


ns, nsi 


na 


ns. Substantively important 


na 








negative effect 




Hecht& Torgesen, 2002 
(quasi-experimental design) 


ns, Substantively important 


na 


ns, nsi 


na 


Waterford Early Reading Program™ 

Hecht & Close, 2002 
(quasi-experimental design) 


ns, Substantively important 


na 


ns, nsi 


na 


Wilson Reading 

Torgesen et al., 2006 
(randomized controlled trial) 


ns, Substantively important 


ns, nsi^ 


ns, nsP 


na 



na = not studied 

ns = not statistically significant 

nsi = not substantively important 

1 . According to WWC criteria, if a program finds a statistically significant effect, then there is less than a 5% chance that this difference is due to chance. The level of statistical significance was calculated by the WWC and, where neces- 
sary, corrects for clustering within classrooms or schools, and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . For the formulas the WWC used to calculate the statistical 
significance, see the Technical Details of WWC-Con clucted Co mputations. 

2. For rating purposes, the WWC considered the statistical significance of the findings and the magnitude of the effect, also called the effect size. An average effect size is the sum of all the effect sizes of the student outcomes in a study in 
a single domain divided by the number of those outcomes. The WWC considers an average effect size across all student outcomes in one study in a given domain to be substantively important if it is equal to or greater than 0.25. 

3. Two interventions each had a study that showed a substantively important negative effect in the comprehension domain (see Fluency Formula and Voyager). 

4. This study reported findings at the cluster level and student-level effect size could not be calculated. 

5. The fluency, comprehension, and vocabulary components of the Wilson Reading Systerrfl were eliminated from instruction at the request of Torgesen et al. for the purposes of the study. For further information about the program imple- 
mented, please see the research and findings sections of the Wilson Reading System® report . 
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Eight hundred eighty-seven studies provided data on 153 
programs and were classified according to the strength of their 
design. To be fuiiy reviewed, a study had to be a randomized 
controlled trial or quasi-experimental design.'' 

Eligibility for review 

Quasi-experiments eiigibie for review include those equating 
through matching or statistical adjustment; regression discon- 
tinuity, and singie case designs are also included. No studies 
based on the regression discontinuity designs were identified for 
the beginning reading review; several single case designs were 
identified. The WWC is currently developing evidence standards 
for regression discontinuity designs and single-case designs. 

The review considered the properties of measurement instru- 
ments, the percentage of students, ciassrooms, or schoois in the 
study sampie that were not inciuded in the reported resuits, and 
any sample characteristics or events that might serve as aiterna- 
tive explanations for the observed effect. For detaiis please see 
the WWC Evidence Standards . 

The research evidence for programs that have at ieast one 
study meeting WWC evidence standards with or without reser- 
vations is summarized in individuai intervention reports posted 
on the WWC website. See http://www.whatworks.ed.qov . So far, 
51 studies of 24 beginning reading programs have met evidence 
standards with or without reservations. The iack of evidence for 
the remaining programs does not mean that those programs 
are ineffective. Some programs have not yet been studied using 
a study design that permits the WWC to draw any conclusions 
about their effectiveness. For some studies, not enough data 
were reported (such as descriptive statistics of the findings) to 
enabie us to confirm statistical findings. 



Rating of effectiveness 

Among the prioritized interventions, each beginning reading 
program that had at least one study meeting WWC standards 
with or without reservations received a rating of effectiveness for 
beginning reading achievement. The rating of effectiveness aims 
to characterize the existing evidence base in a given domain. The 
intervention effects based on the research evidence can be rated 
as having positive, potentially positive, mixed, no discernible, 
potentially negative, or negative effects. 

The rating of effectiveness takes into account four factors: the 
quality of the research design; the statistical significance of the 
findings; the size of the difference between participants in the inter- 
vention and comparison conditions; and the consistency in findings 
across the studies (see the WWC Intervention Rating Scheme) . 

The level of statistical significance was reported by the study 
authors or, where necessary, calculated by the WWC to correct 
for clustering within classrooms or schools and for multiple com- 
parisons. Because of these corrections, the level of statistical 
significance as calculated by the WWC may differ from the one 
originally reported by the study authors. For an explanation, see 
the WWC Tutorial on Mismatch . For the formulas that we used to 
calculate statistical significance, see Technical Details of WWC- 
Conducted Computations . If the average effect size across all 
outcome measures in one study in a single domain is at least 
0.25, it is considered substantively important, contributing 
toward the rating of effectiveness. See the technical appendices 
of the beginning reading intervention reports for further details. 

Extent of evidence 

The WWC categorizes the extent of evidence in each domain as 
small or moderate to large (see the What Works Clearinghouse 
Extent of Evidence Categorization Schemed The extent of 
evidence takes into account the number of studies and the 



1. Thirty-two interventions (involving 36 quasi-experimental design studies) passed the initial screening criteria but were not included in this wave of 

Beginning Reading reviews. These interventions were those that on initiai screening had only one eligible study that met WWC evidence standards with 
reservations (i.e., had the fewest numbers of studies, which also used less rigorous designs). Seven additional single-case studies have dispositions 
pending. The WWC is currently developing standards for the review of single case studies. 
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total sample size across the studies that met WWC evidence 
standards with or without reservations. ^ 

Improvement Index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each domain and 
each study as well as a domain average improvement index 
across studies of the same intervention (see the Technical 



Details of WWC-Conducted Computations^ . The improvement 
index represents the difference between the percentile rank of 
the average student in the intervention condition and the percen- 
tile rank of the average student in the comparison condition. The 
improvement index can take on values between -50 and +50, 
with positive numbers denoting results favorable to the interven- 
tion group. Unlike the rating of effectiveness, the improvement 
index is based only on the size of the difference between the 
intervention and the comparison conditions. 



2. The Extent of Evidence Categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept, external validity— such as the students’ demographics and the types 
of settings in which studies took place— are not taken into account in the categorization. 
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